Skip to content

Conversation

@felixbarny
Copy link
Member

@felixbarny felixbarny commented Sep 15, 2025

This is enabled by adding a new dynamic_templates_params option to the bulk request header. It can be used in combination with the dynamic_templates option to provide parameters that can be used within a dynamic template.

The dynamic template can control which parameters it accepts and can provide a default value in case clients don't provide the parameters in the bulk request. If no default value is provided, and the bulk request also doesn't specify the parameter, the value is kept as it is defined in the template (including braces).

For example, the dynamic template can define a unit placeholder.

{
  "dynamic_templates": [
    {
      "gauge_long": {
        "mapping": {
          "type": "long",
          "time_series_metric": "gauge",
          "meta": {
            "unit": "{{unit}}"
          }
        }
      }
    }
  ]
}

Clients can then set provide a value for the unit in the bulk request header like so:

{"create": {"dynamic_templates":{"metrics.system.memory.usage":"gauge_double"},"dynamic_template_params":{"metrics.system.memory.usage":{"unit": "By"}}}}
{"metrics": {"system.memory.usage": 42}}

Placeholders for which no parameters are specified will evaluate to an empty string.


I chose a {{placeholder}} syntax over {placeholder} to avoid ambiguity as {percent} is actually a literal value that we expect to use which is not meant to be interpreted as a placeholder. This is also consistent with the syntax we use in other places like ingest pipelines (mustache and reroute processor placeholders). However, at the moment, dynamic templates support some parameters with only a single pair of braces: {name} and {dynamic_type}. I've added support for specifying these in double braces as well to maintain consistency (while still supporting the single brace version). Interestingly, we've documented that we already support this syntax: https://www.elastic.co/docs/manage-data/data-store/mapping/dynamic-templates

Use the {{name}} and {{dynamic_type}} template variables in the mapping specification as placeholders.

However, it's not actually true (yet).

This is enabled by adding a new `dynamic_templates_params` option to the bulk request header. It can be used in combination with the `dynamic_templates` option to provide parameters that can be used within a dynamic template.

The dynamic template can control which parameters it accepts and can provide a default value in case clients don't provide the parameters in the bulk request. If no default value is provided, and the bulk request also doesn't specify the parameter, the value is kept as it is defined in the template (including braces).

For example, the dynamic template can define a `unit` placeholder with an empty string as the default value.
```
{
  "dynamic_templates": [
    {
      "gauge_double": {
        "mapping": {
          "type": "double",
          "time_series_metric": "gauge",
          "meta": {
            "unit": "{unit:}"
          }
        }
      }
    }
  ]
}
```

Clients can then set provide a value for the unit in the bulk request header like so:
```
{"create": {"dynamic_templates":{"metrics.foo.bar":"gauge_double"},"dynamic_templates_params":{"metrics.foo.bar":{"unit": "By"}}}}
```
@felixbarny felixbarny self-assigned this Sep 15, 2025
@felixbarny felixbarny requested review from a team as code owners September 15, 2025 09:49
@felixbarny felixbarny added >enhancement :StorageEngine/Mapping The storage related side of mappings labels Sep 15, 2025
@elasticsearchmachine elasticsearchmachine added Team:StorageEngine v9.2.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Sep 15, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@elasticsearchmachine
Copy link
Collaborator

Hi @felixbarny, I've created a changelog YAML for you.

@felixbarny
Copy link
Member Author

We could use a similar mechanism to store the metric description.
However, the meta field imposes limits on the number and size of the entries:

Field metadata enforces at most 5 entries, that keys have a length that is less than or equal to 20, and that values are strings whose length is less than or equal to 50.

For units, this shouldn't be a concern. For descriptions, we could just truncate the description if it is larger than 50. But I heard there's desire to drop or increase the limit for meta. I'd be glad to get pointers for this if anyone has more information on that.

Also adds support for double braces for existing {name} and {dynamic_type} placeholders
@kkrik-es
Copy link
Contributor

It looks reasonable to me. Do we have any performance numbers, wrt impact to ingest throughput? It's noisy, but may be worth an adhoc run to verify that it's not overly slow.

@felixbarny
Copy link
Member Author

Not yet, but it's on my list.

@felixbarny
Copy link
Member Author

Based on the results of an ad-hoc run (environment ci-comp-bench-all-bahkm), there's no regression for the ingestion throughput.


import static org.elasticsearch.index.mapper.TimeSeriesParams.TIME_SERIES_DIMENSION_PARAM;

public class DynamicTemplate implements ToXContentObject {
Copy link
Contributor

@kkrik-es kkrik-es Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@martijnvg do you know who owns this? Do we need another team to review dynamic template changes?

@felixbarny felixbarny linked an issue Nov 27, 2025 that may be closed by this pull request
Copy link
Contributor

@romseygeek romseygeek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@felixbarny felixbarny merged commit 49858d1 into elastic:main Nov 28, 2025
35 checks passed
@felixbarny felixbarny deleted the bulk-meta branch November 28, 2025 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. >enhancement external-contributor Pull request authored by a developer outside the Elasticsearch team :StorageEngine/Mapping The storage related side of mappings Team:Distributed Indexing Meta label for Distributed Indexing team Team:StorageEngine v9.3.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a default mapping template for metrics

5 participants